Improving Postpolypectomy Surveillance Decisions Using an Automated Analysis of Colonoscopy and Histopathology Reports: 2-Center Validation Study

Introduction

Background

Colorectal polyps are a significant risk factor for colorectal cancer (CRC). When polyps are identified during an index colonoscopy, patients may require further colonoscopic surveillance depending on the endoscopic and pathological characteristics of the lesions [1]. Colorectal polyp surveillance improves the detection of CRCs, particularly in high-risk groups defined by specific endoscopic and pathological findings [2-5]. Endoscopy represents a substantial workload in the United Kingdom, with around 2 million procedures performed each year. Polyp surveillance accounts for approximately 15% of the half a million colonoscopies performed annually [1,6].

Postpolypectomy surveillance decisions are particularly vulnerable to inaccuracies. Reported compliance with the updated 2020 postpolypectomy surveillance guidelines varies from 66.2% to 86.5% [7,8]. Poor compliance can often be attributed to surveillance intervals not being updated once pathology results become available [9]. Nonadherence to guidelines can lead to both under- and oversurveillance, with one study reporting 73% of nonadherent patients having earlier-than-recommended colonoscopies [9]. These findings highlight the need for continued efforts to improve guideline adherence in colorectal surveillance and the need for accurate calculation of surveillance intervals based on evolving guidelines.

Several studies have proposed to automate portions of the decision-making workflow, but they frequently rely on applying inflexible rules or pattern recognition to endoscopy and pathology report text (regular expressions) that struggle to accommodate variations in clinical report language. Existing studies are often limited to single-center data or focus on partial tasks such as adenoma detection without extending to full guideline-based follow-up recommendations [10-13]. Wu et al [14] developed an automatic surveillance system for postpolypectomy follow-up that encodes multiple international guidelines and combines free-text endoscopy and pathology reports to assign surveillance intervals. Their approach relies heavily on regular expressions, which are used to identify patients and map extracted polyp attributes. In practice, this design requires a long list of exclusion categories (no pathology result; poor bowel preparation; previous CRC, colorectal surgery, or endoscopic submucosal dissection; nonpolyp lesions; hamartomas; inconsistent endoscopic vs pathological reports; and missing or malformed size information) so that many real-world reports fall back to manual review rather than fully automated surveillance. The system was developed and evaluated in three hospitals in China, which may limit direct transferability to English-language histopathology workflows in UK practice.

Recent studies have evaluated general-purpose large language models (LLMs) for postpolypectomy surveillance interval prediction [15-17]. The larger study by Acharya et al [17] leveraged GPT-4o to generate recommendations from colonoscopy and pathology report impressions using prompted US Multi-Society Task Force guidance. These approaches treat surveillance interval generation as an end-to-end language model task, which leaves clinical decision-making implicit without auditable intermediate representation for clinicians to review.

Our study used a different task breakdown: explicit extraction of specimen-level variables followed by separately implemented British Society of Gastroenterology (BSG) guideline rules. This approach allows for evaluation of the extracted variables and the final interval suggestion. This also allows decision rules to be modified alongside guideline changes independently of the language model. Additionally, we compared automated recommendations with observed care under real-world deployment conditions, looking at patients’ full electronic health record (EHR) history beyond the index colonoscopy.

Aims

The primary objective of the current study was to develop and retrospectively evaluate an implementable and interpretable hybrid decision-making tool that accurately suggests follow-up timings for patients undergoing colorectal surveillance based on colonoscopy and associated histopathology reports grounded in BSG guidelines. We validated our tool on reports from two geographically disparate National Health Service (NHS) trusts in the United Kingdom (Guy’s and St Thomas’ NHS Foundation Trust [GSTT] and Sandwell and West Birmingham NHS Trust [SWB]).

A secondary objective was to assess the tool’s potential clinical utility in routine NHS care. We compared automated recommendations with observed surveillance timing provided by gastroenterologists’ review of full EHRs. This approach enabled us to understand whether automated decision support could reduce under- or oversurveillance and identify limitations of decisions based on the index reports.

Methods

Overview

Figure 1 summarizes the entire pipeline from data access to the evaluation of the AI component. We adhered to the TRIPOD-LLM (Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis–Large Language Models) guidelines [18] and completed the checklist (Checklist 1) ().

‎

Ethical Considerations

Data access and processing at GSTT was authorized by the Guy’s and St Thomas’ Electronic Records Research Interface (Integrated Research Application System [IRAS] ID 257283; Research Ethics Committee ID 20/EM/0112). This approval was effective from June 3, 2023. Data access for SWB was authorized through a data sharing agreement between the institutions in compliance with the General Data Protection Regulation. This work used deidentified, routinely collected clinical data with no direct participant contact. Individual informed consent was not required under the governance and ethics approvals for the data access route. Patients in England can register in the National Data Opt-Out service to prevent use of their confidential patient information for purposes beyond individual care (research and planning).

Data Sources

We extracted colonoscopy and histopathology text reports from GSTT’s EHR system covering procedures performed between January 2017 and September 2023.

Inclusion Criteria

Colonoscopy reports with the mention of the keywords “polyp” or “lesion” were extracted. Corresponding histopathology reports were merged on a per-procedure basis using the EndoMineR package [19] by patient identifier and fuzzy-matched date, resulting in a dataset of over 14,735 reports and 11,538 unique patients. Cases with no pathology information were excluded. Patients under the age of 18 years and patients who had opted out of data analysis using the NHS opt-out system were also excluded. The latter program allows individuals to prevent their confidential patient information from being used for research and planning purposes [20]. The final reports were anonymized by removing all personally identifiable information, including patient names, addresses, and phone numbers. The dataset only contained essential clinical fields for decision-making: endoscopic findings (endoscopy report), macroscopic description (histopathology report), microscopic description (histopathology report), and date of birth. More details about patient involvement can be found in Multimedia Appendix 1.

Training and Test Sets

Ten GSTT reports spanning a range of complexities were selected for model development (in-context learning). For evaluation, we randomly sampled 100 GSTT procedures from unique patients, corresponding to 215 individual histopathology specimen pots from the same date range, excluding the reports included for in-context learning. The test set size was defined as a pragmatic internal validation size given the time required for gold-standard annotations. Table 1 summarizes the demographics of the GSTT test set. For the comparison with current clinical practice, we extracted 127 consecutive GSTT colonoscopy procedures performed between January 2021 and September 2021, along with their associated histopathology reports.

SWB External Validation Cohort

An independent set of 100 colonoscopy and histopathology text reports was obtained from SWB. The same inclusion criteria and deidentification procedures were applied. Date of birth was not shared for the SWB cohort. A CONSORT (Consolidated Standards of Reporting Trials) diagram describing the datasets is provided in Figure 2.

‎

Data Governance and Approvals

As mentioned, data acquisition at GSTT was authorized through institutional board review (IRAS ID257283), under the governance of the Guy’s and St Thomas’ Electronic Research Records Interface. SWB data was provided under a data sharing agreement between institutions. No identifiable patient information left any of the NHS trusts. Patient age was not shared for the SWB cohort, making the algorithm bypass age-dependent conditions.

Task Definition and Annotation

The pipeline aims to extract guideline-relevant polyp features from paired colonoscopy and histopathology reports and then apply the BSG guidelines to generate a follow-up recommendation. A key challenge is that the evidence needed for surveillance is fragmented across sections: endoscopic findings describe the lesions sampled, whereas histopathology reports describe multiple specimens (“pots”) with macroscopic and microscopic text that must be linked to determine which pots correspond to polyps, which represent nonpolyp tissue, and which findings are unresected or described only endoscopically. The GSTT test set was annotated by a committee of 3 clinical experts to establish the gold standard. For each procedure, annotators reviewed the endoscopic findings and the histopathology macroscopic and microscopic descriptions to derive the number of histology pots and extract pot-level variables listed in Table 2. “Premalignant” and “advanced” polyps were defined according to the BSG criteria [1]. Disagreements between annotators were resolved through consensus. When consensus could not be reached, a fourth clinical expert (SZ) resolved the discrepancy. The SWB test set was annotated by a single gastroenterologist (SZ).

Approach and Model Selection

Overview

Our objective required extracting multiple interdependent variables (Table 2) from heterogeneous free-text colonoscopy and histopathology reports. Pot-level alignment and cross-referencing were difficult to model reliably using simple pattern-matching rules. A purely supervised approach would require large-scale, pot-level annotations to generalize across report templates and anatomical sites. In preliminary experiments, a fine-tuned encoder-based baseline did not generalize well under the available annotation budget. We therefore used LLMs as flexible extractors that could be constrained to produce structured outputs with minimal labeled data. The design of this study explicitly separates information extraction from clinical decision-making. The LLM was not used to infer risk categories or recommend intervals. The final surveillance recommendation was computed by a deterministic, auditable, guideline-encoded rule engine.

Model Selection

Given typical on-premises hospital hardware constraints, we evaluated which open-weight instruction-tuned LLM size was sufficient for this task without additional supervised fine-tuning. We compared multiple models (Microsoft Phi-4, Phi-4-mini-instruct, Llama 3.1-8B-Instruct [Meta AI], Falcon3-7B-Instruct [Technology Innovation Institute], and Mistral-7B-Instruct-v0.3 [Mistral AI SAS]) under an identical prompting, decoding, and parsing configuration. We report the surveillance accuracy for each model in Multimedia Appendix 2. We used Microsoft Phi-4 [21] (Hugging Face) at revision 932b33c for the actual pipeline.

Hardware and Pipeline Setup

All LLM inference was executed on local graphics processing unit (GPU) hardware (NVIDIA DGX-1 server running Ubuntu 20.04, configured with eight 32-GB V100 GPUs and 512 GB of RAM). We loaded the model with 4-bit quantization and initialized the model with automatic sharding across available GPUs. Generation was performed in batches to improve throughput. Decoding was deterministic, with a temperature set to 0. Each pipeline step used a configured maximum number of new token budget. When output parsing failed, the pipeline retried generation with an increased token budget per attempt.

Multi-Prompt Strategy

Breaking down complex information extraction tasks into smaller subtasks has proven to improve the performance of LLM pipelines [22]. The task at hand was complex by nature as relevant information is fragmented across report sections and multiple specimens (pots) must be aligned before guideline features can be extracted. We therefore broke down the pipeline into four sequential prompts that progressively (1) segmented histopathology text pot by pot, (2) appended endoscopic findings to the corresponding pots, (3) normalized each pot into a structured schema, and (4) extracted guideline-relevant attributes. Figure 3 illustrates this progressive transformation for a synthetic report.

‎

Rule-Based Logic

The structured outputs produced by the final LLM prompt were parsed into histopathology pot-level variables (Table 2) and passed to a deterministic rule-based algorithm that encoded the guidelines. The algorithm first raised an alert if cancer was detected, then filtered out nonpolyp specimens; aggregated polyp counts and sizes across all pots; identified premalignant and advanced polyps using guideline-aligned criteria; and, finally, output a surveillance recommendation according to the BSG, Association of Coloproctology of Great Britain and Ireland, or Public Health England risk categories (no surveillance, 3-year surveillance, 5-year surveillance, or site check for large nonpedunculated polyps with uncertain excision). If age was missing, the algorithm ignored age-based conditions and defaulted to the non–age-specific guideline branches.

The pseudocode functions “IsPremalignantPolyp,” “IsAdvancedPolyp,” and “IsHighRisk” implemented guideline definitions of premalignant polyps, advanced polyps, and high-risk patients. In summary, premalignant polyps included adenomatous polyps and clinically relevant serrated lesions (excluding diminutive rectal hyperplastic polyps), and advanced polyps were defined by size and/or dysplasia according to the UK guideline thresholds. High risk was assigned when either 2 or more premalignant polyps were present, including 1 or more advanced polyps, or 5 or more premalignant polyps were present.

The algorithm compared the endoscopic polyp count with the number of histopathology specimens to detect specimen fragmentation and reduce pot-level size and count estimation bias. The algorithm preferentially used the endoscopic size as pathology-reported measurements may underestimate true size due to specimen shrinkage after formalin fixation [23]. The pseudocode for the rule-based algorithm can be found in Multimedia Appendix 3.

Model Evaluation

Overview

Most studies on LLMs in health care focus on accuracy. However, accuracy alone is not enough for clinical integration: reliability, completeness, and deployment feasibility are equally important. We evaluated our chain-prompting pipeline across multiple dimensions aligned with best practices in health care LLM evaluation [24-26]. Some commonly used evaluation criteria such as human preference do not apply to this study, which is focused on structured information extraction rather than free-text generation.

Accuracy and Comprehensiveness

Accuracy was assessed by comparing the model’s extracted values to gold-standard annotations for each category. We evaluated information extraction at the pot level by comparing the model output against clinician gold-standard annotations. For single-value fields, including number of pots, specimen type, excision completeness, location, number of specimens—findings, and number of specimens—macroscopy, correctness was defined as an exact match to the gold-standard label. For the size field, order was ignored, but all size entries had to match the gold-standard annotations. Diagnosis was evaluated using a structured “diagnosis signature” derived from the microscopic description consisting of information about the lesion type (tubular adenoma, tubulovillous adenoma, serrated lesion, or hyperplastic), dysplasia grade (none, low grade, or high grade), and invasive cancer (present or absent). A diagnosis was scored as correct only if all components of the signature matched the gold-standard annotation for that pot. Negation was evaluated by comparing the set of clinically critical negated concepts extracted for each pot to the gold-standard set.

Comprehensiveness was evaluated at the report level as the proportion of gold-standard fields whose values were extracted exactly correctly. Because incorrect or missing values are counted as false negatives, this metric is equivalent to per-report, field-level recall:

Comprehensiveness = (number of gold-standard fields extracted exactly correctly)/(number of gold-standard fields)

We averaged this score across reports to obtain an overall measure of the model’s ability to extract all the expected fields. We also report the F₁-score per category, capturing both precision and recall. The 95% Wilson CIs were calculated for overall recommendation accuracy and class-specific precision and recall.

Robustness

Robustness was assessed by evaluating the model’s performance on a perturbed version of the test set, inspired by established methods in the literature [27-29]. To simulate real-world typos, we introduced controlled typographical noise, including adjacent letter swaps, random character deletions, insertions, random word splits, and letter case alterations. Synonym replacement was not performed as it is rarely encountered in real-world practice (eg, “adenoma” has no exact lexical variant). Key terms such as “polyp,” “adenoma,” and “dysplasia” were intentionally corrupted at least once per noisy test set to test the model’s resilience to critical colorectal polyp terminology distortions. We generated multiple noisy test sets with increasing perturbation levels (1 to 5 per report) and compared their F₁-scores against that of the original clean set. Examples can be found in Multimedia Appendix 4.

Deployment Feasibility

Deployment feasibility was assessed by measuring computational requirements, including the number of model parameters and memory use.

Retrospective Analysis of Current Practice

In addition to per-report extraction metrics, we performed a retrospective, real-world eligibility study where we compared the automated system recommendations and observed care to a clinician-derived reference based on full EHR review. The automated system used the index colonoscopy and associated histopathology reports, with age where available. This analysis aimed to evaluate how the automated surveillance recommendations would align with guideline-based care when full longitudinal patient information was available.

A single gastroenterologist (SDM) reviewed the complete EHR of all the patients in the retrospective test set (n=127). The clinician determined the gold-standard guideline-concordant surveillance interval for each patient accounting for but not limited to previous CRC or colorectal surgery, prior large or advanced polyps, hereditary or other high-risk conditions, and comorbidities or contraindications that would alter surveillance timing. We then compared 3 intervals: the clinician gold-standard interval from full EHR review, the interval recommended by the automated system using only the index colonoscopy and associated histopathology reports, and the observed care interval based on real-world scheduling (a completed follow-up colonoscopy or a future booking recorded in the EHR). Observed care was assessed using all information available in Epic. A completed procedure, future booking, or absence of surveillance were classified looking at all post–index colonoscopy data available in Epic. Reference assessment was blinded to model output. This analysis assessed the extent to which this system could improve alignment with guideline-concordant surveillance and, therefore, its potential impact on clinical practice. Full EHR review also identified patients whose management required a clinical pathway beyond the postpolypectomy recommendations generated by the system.

Results

We report per-field extraction performance at GSTT, surveillance recommendation performance at GSTT and SWB, robustness to typographical noise, an analysis of the trade-off between model size, and on-premises feasibility and performance. Unless otherwise stated, results correspond to the final Phi-4 multistage prompting configuration.

Extraction Performance

Table 3 shows that the extraction of key fields for the final surveillance decision was highly reliable (mean comprehensiveness 0.968, SD 0.112), with close to perfect accuracy for location, type of specimen, and histopathological size (0.981, 0.995, and 0.995, respectively). The model also showed high accuracy for diagnosis (0.981) and excision completeness (0.949). The endoscopic size, number of specimens, and extraction of unresected polyps showed the lowest accuracy (0.935, 0.935, and 0.903, respectively).

Surveillance Recommendation Performance and Cross-Site Generalization

Table 4 summarizes the performance of the decision support system for surveillance recommendations at GSTT and SWB. The system achieved an overall accuracy of 0.96 at GSTT and 0.93 at SWB, with strong performance for site check (F₁-score of 0.93 at GSTT and 0.91 at SWB) and no follow-up (F₁-score of 0.98 at GSTT and 0.97 at SWB). Cancer alerting exhibited perfect performance (F₁-score=1.00) for both sites.

Robustness

Detailed robustness results across perturbation levels are provided in Multimedia Appendix 5. Across progressively perturbed test sets (1 to 5 perturbations per report), F₁-scores decreased only modestly compared with the clean test set, indicating that the pipeline is robust to realistic typographical noise and minor formatting artifacts. In particular, performance on the number of pots, type of specimen, and diagnosis remained stable, whereas size extraction showed the greatest sensitivity to perturbations.

Retrospective Analysis of Current Practice

Using gold-standard guideline-concordant intervals derived from full chart review, we audited current practice within the department for 127 patients in the test cohort and compared it to the system’s suggestions. Overall, only 59.1% (75/127) of patients were booked in line with the recommended surveillance interval. A further 15.7% (20/127) deviated from guideline-concordant care: 5.5% (7/127) were potentially oversurveilled (either booked earlier than recommended or scheduled for an unnecessary colonoscopy), and 10.2% (13/127) were potentially undersurveilled (6/127, 4.7% were booked later than recommended, and 7/127, 5.5% had no surveillance booked despite needing one). A further 25.2% (32/127) were managed via alternative pathways (CRC or other colonoscopy pathways), where surveillance scheduling was driven by additional clinical findings (Table 5). Even excluding these alternative-pathway cases, 21.1% (20/95) of patients in routine care had potentially missed or unnecessary surveillance relative to guideline-concordant recommendations. In contrast, our system provided a consistent, automated, and guideline-based recommendation for every patient based on structured information from index colonoscopy and histopathology reports. Overall, the decision support tool improves alignment with guideline-concordant care (Table 5), particularly by reducing undersurveillance. These findings highlight a real opportunity to identify patients at risk of undersurveillance and potentially reduce unnecessary procedures, supporting efficient use of endoscopic resources.

Discussion

Principal Findings

This study introduced a decision support tool that combines open-weight, medium-sized language models for extraction of guideline-relevant information with a deterministic rule-based algorithm for postpolypectomy recommendations. Focusing on implementation encouraged the use of a computationally lightweight LLM to only perform report standardization and information extraction. We demonstrated that such a decision aid system presents an opportunity to improve guideline adherence by conducting a retrospective chart review using patients’ full medical histories, showing that routine surveillance sometimes occurs later than guideline-recommended intervals. We identified cases with no indicated surveillance booking, highlighting a concrete opportunity to strengthen safety netting and reduce missed surveillance in routine care.

Acharya et al [16,17] evaluated direct LLM-based surveillance interval assignment in an initial 100-case feasibility study and subsequently in a 2-institution cohort of 1000 cases. The latter study reported a mean GPT-4o accuracy of 94.6% across 10 runs. Our recommendation accuracy was 96% at GSTT (95% CI 90%-98%) and 93% at SWB (95% CI 86%-97%). These independently obtained results cannot be compared directly as the studies differ in guideline systems, populations, output categories, and evaluation scale. The later study by Acharya et al [17] also provides evidence from a substantially larger validation cohort.

In the larger study by Acharya et al [17], accuracy was lower for cases containing 4 or more polyps than for those with 1 to 3 polyps. This finding highlights the importance of evaluating complex cases. Our framework separates specimen-level extraction and aggregation from deterministic guideline application, which facilitates granular evaluation at the individual polyp level.

Strengths

There are several key strengths to this study. While general-purpose LLMs are increasingly used for a wide range of tasks, our work focuses on a specialist clinical setting in which smaller open-weight models are applied to narrow, auditable subtasks. This design choice supports transparency and increases clinician confidence by making the extracted information and decision logic available for inspection. We divided the overall task into components that closely mirrored clinical practice: collating endoscopy and histopathology reports, extracting information pot by pot, aggregating findings across pots, and deriving a surveillance recommendation. It is important to highlight that the language model was only tasked with extracting and normalizing information already present in the reports and did not generate the surveillance recommendation. The LLM did not generate any clinical decision-making, and the surveillance decision itself was made by a rule-based algorithm anchored in the guidelines, which can be audited and updated independently as guidelines change. By separating extraction from decision-making, the framework could also be adapted to other gastrointestinal surveillance pathways by modifying the extraction layer and introducing condition-specific rules.

The evaluation of the language model information extraction was performed using a range of metrics (accuracy, comprehensiveness, F₁-score, robustness, and deployment considerations) that provide a more complete picture of performance than accuracy or F₁-score alone. The SWB evaluation provided evidence of performance in a second environment with differing documentation styles and reporting practices subject to its small sample size and the restriction concerning age-dependent rules.

This study deliberately focused on small- to medium-sized language models to reflect typical on-premises hardware constraints in NHS settings. While some hospitals can process clinical text via external LLM APIs, this capability varies widely across institutions due to governance and privacy requirements, and on-premises inference is generally preferred.

Limitations

Future work should include larger and more geographically diverse sites to better assess robustness and generalizability.

The retrospective analysis highlighted that a subset of patients (n=32) required referral to alternative pathways (eg, inflammatory bowel disease or ulcerative colitis), which is currently out of scope for this study. This reinforces why this study introduced a decision support tool rather than a fully automated system: clinicians sometimes need to combine multiple guideline pathways and clinical context to make the correct decision. The SWB and the retrospective audit annotations were provided by a single gastroenterologist, which may introduce reference standard bias.

The two 100-procedure test sets provided limited precision, particularly for minority classes. Larger cohorts with diverse representations of rare outcomes are needed to draw broader generalizability conclusions. The small number of ALERT cases resulted in wide CIs.

The GSTT test set was older (mean 69.4, SD 9.35 vs 57.3, SD 14.50 years) and had a higher recorded proportion of procedures involving White patients (93/100, 93% vs 7625/14,735, 51.7%). Interpretation was also affected by the larger combined “other or not specified” category in the source cohort (4610/14,735, 31.3% vs 3/100, 3%). These differences limit representativeness and may reflect differences in the sampled clinical population. The small numbers in some demographic groups precluded formal subgroup performance analyses across sex or ethnicity. Instead, we report overall performance and highlight fairness as an important direction for future work, particularly as larger and more diverse validation cohorts become available.

Finally, although we compared models of different sizes, the final configuration used a 14 billion–parameter model with 4-bit quantization on the reported hospital server. While our institution has the necessary infrastructure, we acknowledge that this may not be the case in other settings. However, the modular design and code available upon request allow the language model component to be swapped for smaller or fine-tuned alternatives, enabling adaptation to local computation constraints.

Conclusions

This study demonstrates that a hybrid automated system can generate accurate and auditable postpolypectomy surveillance recommendations from paired colonoscopy and histopathology free-text reports. The framework enables inspection of intermediate information and rule application. External validation across two NHS trusts suggests that the approach could generalize across different reporting environments. The retrospective comparison with routine care suggests potential improvements over current surveillance practice in specific patient groups. It also identified 25.2% (32/127) of patients requiring alternative clinical pathways, highlighting the importance of broader patient history. Future research could therefore investigate whether integrating longitudinal clinical information helps distinguish patients eligible for routine postpolypectomy surveillance from those requiring alternative pathways. Future research could also explore whether smaller open-weight language models can be used to extend this framework and more broadly support gastroenterology surveillance pathways under realistic deployment constraints.